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[57] ABSTRACT 

A versatile video transformation device and adaptive image 
processing methodology thereof to digitally scan convert, 
that is, reformat TV raster scan video and particularly high 
definition (HD) and/or digital DTV (particularly those for 
example in 1920xl080i or 1280x720p format) video data 
and associated synchronizing signals, for the purpose of 
making present standard television sets compatible at low 
cost with the latest advancements in free HD whilst allowing 
a multitude of other DTV programs and ancillary data to fill 
to a greater extent the remaining channel allocations by FCC 
to TV broadcasters. 

19 Claims, 8 Drawing Sheets 
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VERSATILE VIDEO TRANSFORMATION evolution reveals the perennial problems needing solution to 

DEVICE meet the resolution revolution that now requires brighter, 

FIELD OF THE INVENTION smarter picture processing to live up to HDTV promises of 

35 mm quality. 

The present invention is directed towards versatile video s Naturall thc ^ h of thc icturc fa blurrcd b jy 

transformation device and adaptive image processing meth- cameras ^ ^ ^ tfack tQ k ^ ^ ^ q{ ^ 

odology thereof to digitally scan convert tha l is reformat mQy{ ^ ^ afc ccn(cr rf attracdon sucfa fls m ^ 

TV raster scan video and parucularly high definition (HD) skatef * f ^ ^ ^ 5asket5all uck etc . From 

and/or digital DTV video data and associated synchromzing c ^ huffian factors data obtaincd from ^ and 

signals, for the purpose of making present standard tetevi- , 0 « movk ^ converters (Rand Corp., RCA Corp., 

sion sets compatible at low cost with the ^latest advance- F ^ chM Camera et al) a should mQve {hi h ^ 

ments in free HD whilst allowing a multitude of other DTV standard ^ fieJd Qf yfcw ^ 4 ^ ^ of mofe bu| 

programs and ancillary data to fiUto a greater extent the nQ ^ ^ Qne stC0Q± Qne secQnd ^ time for 

remaming channel allocaUons by FCC to TV broadcasters. refreshing 60 fields ^ ^ 3 4 fixations of 5x5 degrees at 

BACKGROUND OF THE INVENTION ^ 0.3 second assuming 480 scan lines each subtend arc minute 

Present TV sets are not capable of using HDTV, but rather defined as 20/20 vision, 

are limited to a fixed-format and are incompatible with Thus those conclusions that an object ought not to move 

different TV raster formats such as that provided by HDTV. more than l-to-4 scan line in Veoth second field is consistent 

While an amount of versatility (and energy -efficiency) has with video compression ratios of 30-120 since 1.7%-0.42% 

been provided by U.S. Pat. No. 4,361,785, the disclosure of 20 or less is changed in either or both axes, 

which is incorporated herein, it is desirable to improve and The prior art of monochrome video scan conversion was 

expand it, particularly, with regard to standard TV sets. considered to be a linear system using analog and later 

The inexpensive utility of versatility for single frequency digital means with predictable performance. One of the 

standard TV is especially desirable now because of the ^ earliest scan converters involved a TV camera scanning a 

availability of HDTV and DTV. radar PP1 plan position indictor whereby the video values at 

In the past, when color TV broadcasts became available, "rho-theta" (range and angle) polar coordinates driving an 
existing monochrome "black & white" TV sets, at the time, oscilloscope display with long persistence phosphor were 
could not display the added dimension of color. Viewers transformed to the x,y Cartesian coordinates of the horizon- 
could, however, still see in monochrome sets new programs 30 tal and vertical deflection in TV camera feeding a synchro- 
broadcast in color because the novel chrominace signals mzed TV display. It and subsequent scan converters also 
conveying the hue and saturation (color purity) attributes used to convert diagnostic ultrasound and sonar signals to 
were rationalized and compressed into the then existing TV images mechanized the coordinate transformation 
video bandwidth without substantial interaction with the matrix called the Jacobian in Calculus texts, 
luminance. While the present invention is not directed 35 Generally, when higher resolution ultrasound or radar 
towards improving the TV set circuitry per se, in so far as indicators or TV rasters eg. 1225 lines EIA-RS-343 are scan 
DTV system eliminates many visual impairments eg. inter- converted to standard 525 lines EIA-RS-170, 330 severe 
line flicker, sync jitter, moire, raster line structure, multipath Moire patterns form by the beat-frequency between the 
ghosts, chrominace distortions etc. the picture quality of horizontal line frequencies if the readout beam was not 
such TV sets should be markedly improved. This will be 40 sufficiently defocussed to span several input scan lines, 
accomplished while, most importantly, addressing the Preferably an anti-Moire digital filter or dither analogous to 
broadcaster's desire to allocate, as much as possible, other analog "wobble" signal can modulate the scan converter 
programs and data communications within the channel input PPI deflection as a function of rho, theta to widen the 
assigned to it by the FCC. spokes as it were near the rim when nearly parallel to 

In this regard, the FCC has allocated the traditional 45 horizontal scan lines readout, 

broadcasters a new channel for DTV which can include A patent in the early 1970's granted to Don Dudley of 

"multicasting" several simultaneous programs with condi- Dataplex Inc. explained how 1225 line HDTV could be 

tional access for a nominal fee that would offset the costs of broadcast in place of redundant fields via standard 525 line 

new equipment needed to broadcast DTV and free HDTV TV In principle, that is what HDTV does today in trans- 

and also provide better competition with costly cable TV and 50 mitting the differences between fields with more robust 

satellite TV. algorithms for removal of redundancy subject to acceptable 

Ironically, the effective expansion of TV channel band- rates of distortion and the empirical 70% Kell factor or 

width capacity wrought by removal of redundancy in video actually 2/pi-64% modulation transfer function (MTF) at 

compression techniques for HDTV appears directed towards Nyquist sample limit. 

the of viewers' desire to receive more channels due to the 55 Another early scan converter involved the simplistic 

quality of programming, or lack thereof. The improved reduction of 625 TV lines 25 frames/second in Europe to 

quality of higher resolution pictures — that are not really 525 TV lines 30 frames/second in USA by cropping off 100 

discernible at present distance between set and viewer, is lines, including at times the forehead of the televised person, 

desirable and possibly only equally important to the viewer Subsequently, instead of cropping 50 lines from the top and 

than the ability to receive additional channels. Accordingly, eo bottom, 4 of every 25 lines were dropped out or "decimated" 

it appears that a need exists to provide a higher quality since 625/525=25/21. Preferably, as DTVFSC demonstrated 

picture as that now made available by HDTV while allowing in 1985, for each and every one of 21 readout lines falling 

the broadcaster to utilize to the greater extent, the remaining between 2 of 25 lines weighting factors proportional to their 

portion of the channel allocation for additional programming separations enabled the interpolation of video values 

or "data casting". 65 between the 625 lines and smoother decimation to 525 than 

As to the first objective, which, if addressed properly, simplistic drop outs. Likewise for example, 720p*y3= 

allows the second objective to be met, the history of TV 1080l*V9-480. 
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In addition to spatial artifacts of aliasing from subsam- aging circuits. In this regard, briefly, the present invention 

pling and of contouring from quantization in digital scan effectively circumvents the luminance blurring and chromi- 

conversion, the different frame rates created temporal prob- nance cross talk at edges due to averaging or due to low pass 

lems with moving images analogous to the stroboscope mters at half me sampling frequency and dithers spatial filter 

effect of backward wagon wheels in sampling 25-30 frames/ 5 sparkle to smooth flicker. 

second. The modem solution by Silicon Graphics et al. is It is also an object to provide video transformation of 

known as "morphine" is disadvantageous since its temporal DTV noninterlaced, so called progressive scan to standard 

interpolation/decimation between fields would require interlaced raster format ^and conversely transform interlaced 

brute-force RAM or excessive software "bloatware" for fields to noninterlaced frames for PC monitors and printers. 

MPEG-2 decoding io 11 * s a forth 61 " object to provide enabling technology and 

n c j • ■* i * j . /daial „ local storage for compact video router transformations to 

Before digital random access memories (RAM) became * . 

. 6 . i ^ • f.„j „ t-w f^nmam hiKo allow and to promote individually addressed video trans- 

inexpensive and commonplace, mstead of a TV camera tube . . t *\ . . / . . . 

r . pnT * j * ■ * u r*rrr t<.~ . missions through quasi cross-point switches connecting any 

scanning one CRT picture and driving another CRT, Texas , _ A r , c , 

j . TI , f n- a ~iu~ r number of video sources to any number of video desuna- 

Instruments, Tektronix, Hughes Image Devices and other ^ ' 

companies produced scan converter tubes with electronic 15 „ , , . . ,. ... . % % 

input (write) and output (read) video whose associated lt 1S a vet *J«*™ m ° ™ * m ^ an ^ bll f 

horizontal and vertical deflection coils were independent in U P to 65Kx65K pixels of MPEG so as to provide the 

so called "double-ended" tubes simultaneous read and write. sensation of realism. 

They were time-shared in cheaper "single-ended" tubes as in Lastly, it is the ultimate object of the invention to provide 

RAM. Most of these CRT scan converters were eventually 20 for enhance quality TV resolution as that provided for by 

replaced by digital image processors or TV "frame grabbers" HDTV while reserving the remainder of the channel allo- 

using RAM in a "brute-force" manner because of the cation for additional programs and ancillary computer data. 

Nyquist sampling limit since RAM was so inexpensive. The present invention provides for the foregoing objec- 

Just as the Taylor Series representation in the time domain tives by providing for a means to receive, decode and scan 

of the video signal by summation of its time derivatives is 25 convert 1920xl080i or 1280x720p HDTV or other DTV 

the mathematical basis for interpolation or reading video fonnat on standard TV sets while substantially enhancing 

values between the scan lines so also the Fourier series display lhereof - 11 provides for sampling and filtering of the 

representation in the frequency domain by summation of definition TV images for scan conversion without 

harmonics of complex sinusoids is the basis of video com- l yP ical crosscoupling of chrominace and luminance so as to 

pression in terms of DCT Discrete Cosine Transform that is 30 s P arkle on lowcr resolution TV sets while substantially 

the real part of the Fourier Transform. Accordingly, the maintaining the input detail subject to viewers complex 

video transformation may be mechanized in either the time spatial/temporal flicker tolerance. It does this by a dithering 

domain as in all known scan converters or in the frequency of raster origin offsets in zigzag fashion within 3x3 or 2x2 

domain as will be discussed. arra y of P™ 15 for scan conversion into one pixel readout. 

- , j, j t 35 This allows for an HDTV quality picture to be broadcasted 

While in the past vanous approaches addressed changes j j u 5 -a*u ♦ n™ 
in broadcasting forma* and changes in the industry, pTes- ° n * format *«™8 a reduced bandwidth so s to allow the 
ently what has not been addressed successfully is the ability broadcas er to have a greater portion of the remaining 
to provide for enhanced quality TV resolution as provided C ^ ^ °T ^^^^ ST 
for by HDTV and DTV in conventional TV while reserving ^ J° uUllze ,he HDTV °\ f ° rma broadcasted, 
as much as possible or desired of the remainder of the 40 T*» P' esent invention also provides for renewable con- 
channel allocation for other programming and/or ancillary ditional access to demultiplex multicast program and anal- 
data. This is something that is desired by both the viewers lal y data compressed in packets within one DTV channel 

.... j allocation. It enables standard TV sets to become compatible 

and the broadcasters. ^ ^ juyrv and flexible formats of PC monitors and 

SUMMARY OF THE INVENTION « hard copy printers. Moreover, its electronics may be incor- 

r , . . ... . , porated within newer TV sets to minimize redundancy of 

It is an object of the present inventior »to provide *e £ electronics may be incorporated within a 

fullest fidelity, video data integrity of HDTV within the . circuit cni 

constraints of standard TV sets which is cost effective. g ^* 

It is a further object to provide externally image quality 50 BRIEF DESCRIPTION OF THE DRAWINGS 

enhancements eg. "sparkle" of broadcast formats for HDTV Thus by the present invention its objects and advantages 

so as to optimize the flicker-free field of view or sparkling will be realized, the description of which should be taken in 

clarity of TV and provide for full view of the uncropped conjunction with the drawings wherein: 

HDTV picture or full use of present TV screen. pio. 1 is a general block diagram of the present invention 

It is also an object to provide buffer storage for refreshing 55 showing conventional "A/B" switch between Satellite or 

of compressed images and for video transformation without CABLE TV & FREE TV ANTENNA feeding into WTD 

having to decode and again compress and re-encode the STB (set-top-box) which feeds the TV via optional VCR, not 

HDTV. In this regard, briefly, the present invention provides yd digital or high definition. The finer HDTV "virtual 

scan conversion in the frequency domain using the equiva- screen" as it were behind the coarse TV screen illustrates 

lent Fast Fourier Transforms (FFT) of time scaling, shifting 60 how the comparable 1920xl080i or 1280x720p actually 

and of video derivatives for interpolation/decimation "SPARKLE" using zig-zag ordered raster dither of origin 

because N nodes are efficiently reduced from NxN opera- offset to exploit the space -time bandwidth of viewers eye- 

tions to NxLn(N) and because it exploits benefit of com- brain. Note the well established fixate rate of 300x300 arc 

pression ratio which is proportional to resolution. minutes/300 milliseconds integrates 18 progressive frames 

It is a further object to provide without objectionable 65 vs 9 1080i frames. Thus the progressive HD's radical-2 

aliasing or flicker remarkably higher resolution than is improvement makes the perceived resolution and signal/ 

ordinarily expected by Nyquist sampling theorem and aver- noise nearly identical for 720p-1080i. 
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FIG. 2 illustrates a comparison of raster formats allowing Full TV Channel Capacity 6 MHz (VGA) 640x480 

interlace 1920xl080i thru 3x3 filters or progressive 1280x Radical-2 Resolution 12 MHz 2Di Dot interlace 

720p thru 2x2 filters plus Multicasting of 4 other ancillary NTSC m ^ 4 2 MHz 443^35 (typical xv set) 

TV data or scrolled datacasting which fit conveniently in the KrTX:r . - „ - A A/fl , - fl/1 ~ QQ A/r , D \ 

bottom (or top) 25% otherwise blanked because the HDTV 5 KTSC m rolor 3 6 MHz 384x288 <*yP lcal VCR > 

aspect is 25% shorter than standard TV height. Monochrome (256 gray levels) also 320x240 dots 

FIG. 3 is a detailed functional block diagram of the Simplest bilevel, black/white TV field 320x240 dots 

versatile video transformation including an in situ CPU, The HDTV "virtual screen" lattice in FIG. 1 vibrates 

renewable conditional access key, advertisers credit/ behind the coarse TV screen as it were so that comparable 

coupons and by way of example the 2x2 and 3x3 operations 10 resolutions of 1920xl080i and 1280x720p are squeezed 

on 1280x720p and 1920xl080i HDTV which are two of the through 3x3 and 2x2 filters as FIG. 4 demonstrates. They 

most likely of the 18 flexible formats of DTV processed by "SPARKLE" via zig-zag ordered raster dither of origin 

the WTD offset by exploiting space-time bandwidth of viewers eye- 

FIG. 4 is a functional diagram of block 41 of FIG. 1 brain It is important to realize the well established fixate 

illustrating the origin offsets of ordered raster dither provid- 15 ' ate of degrees per 0J second permrts 18 progressive 

ing the SPARKLE feature of the present invention for the £ be mte 8 rated *' «° H z vs ? i nterkce frames a V 30 

two most likely HDTV rasters, 720p and 1080i. It also » z ^progressive HD s radical-2 unprovement makes 

. . £ t . 0 ^ r , r - „ i the perceived resolution and signal/noise comparable, not to 

presents a graphic representation of the 2 dimensional *T . . * -n 

dot-interlace (2Di) readout raster frame of 640x480 partial- mentloa . less ™»« arti&cts ™* ™>p *» d «« ^ary 

larly for TVs and VCR's with about half the 6 MHz 20 dat^asUng an^l multicashng of several program^ 

• j c T ,^ A A i ■ l o * _ * * nrv o- Therefore the crisper, wider visual impact of FCC, ANSI/ 

bandwidth for VGA, A radical-2 improvement in 2Di effec- owo __ A JL' rc « Tir „ f in «n 

. w • *k u it « * ^ SMPTE, CCIR, ASTC, ISO, IEC specification of 1280x 

live resolution results on axis from the phase alternated 1ft on' • *. i f u • j u 

M x/u ff . ill 720p and 1920xl080i picture elements when viewed where 

11.93 MHz offset sample clock. . / . f 4Unw 4 , , 

r just resolveable suggests that HDTV screens must be viewed 

FIG. 5 illustrates by simple example of interpolation and M from , A _ y ^ the present average or else be at least 

decimation of the present invention of HDTV 720p lines by times as hi h ^ standard (NTSQ TV picture tubes. 

% and also 1080i lines by % to 480 lines when the viewer DTV , S rcduced transmission impairments lets viewers come 

chooses to fill the full height of the TV with HDTV, knowing doser and i(s subna nosecond sync jitter eliminates interline 

25% of the HD picture width is necessarily cropped offsince flicker a$ a precondition for SPARKLE function. The effec- 

the 1 ^ HDTV aspect is ^rds wider than standard V 3 aspect. 3Q ave vibration or pseudo random dithering of the high 

DETAILED DESCRIPTION OF THE ^ tion " virtual ^ 0U ^ h ?? ^ resoludon 

PREFERRED EMBODIMENTS TV screen apertures is further discussed below. 

A comparison of raster formats is set forth in FIG. 2. In 

Turning now more particularly to the drawings, FIG. 1 this regard the upper raster format 2A shows the proposed 

shows the general set-top-box of the present invention in this 35 raster format for HDTV 1920xl080i and 1280x720p illus- 

regard. The utility of versatility is especially desireable trated for example on a 30 inch high screen having a width 

because of the present lingering debate and uncertain future which is l % times the height and 33% wider than a standard 

of 1920x1080 interlaced lines preferred by some TV broad- TV set of equal height which is illustrated in FIG. 2B. 

casters over the comparable 1280x720 progressive scan FIG. 2B shows two ways of fitting the HDTV signal into 

favored for action sports, multicasting, datacasting and by 40 the standard size, by either cropping the HDTV width or 

personal computer companies because of high speed low blanking 25% of standard TV picture height. However 

cost data networking. cropping has the disadvantage of losing 25% of the picture 

The TV input signal may be received from either broad- and shrinking wastes 25% of the 480 standard raster lines or 

cast antenna 2, from cable TV 4 or satellite which is height. 

transformed and output via optional VCR 8 to the TV set 12 45 At the same viewing distance, the most noticeable differ- 
preferrably as video signal or on channel 3 or 4 RF radio ence between the new TV & old is the wider HDTV screen, 
frequency signal (60-66 or 66-72 MHz). A cable TV or which is 16 /& times the height or 4 A times the old NTSC width 
satellite box 6 may be provided and has its own RF tuner so which is Vi times the same height, whether 1080 lines or 480 
it is also an objective of this "set-lop box" 10 to ultimately lines. As illustrated in FIG. 2B instead of dropping out 5 of 
avoid the expense of a third or fourth tuner in the system and 50 every 9 HDTV lines per frame, the present invention inter- 
thereby be even lower cost. polation and decimation into 4 NTSC lines interlaced 
He present invention is directed to converting digital and b 5 tween 9 HDTV of 2 fields is smoother and preferred 
high definition TV to standard TV with fullest fidelity and * mo "? n 15 n °J se f5 d 11113 M1 standard hei 6 ht " 
image integrity desireable in accordance with ergonomic , ln f 1T pa ^° a ^L™! 
definition for 20/20 vision, i.e. a arc minute of resolution 55 ° f exara P °> for H = 100 ™»«f °L d wdlh * 1333 * •? d 
(291 microradian) and also in accordance with economic new ^ o™*™ l ° Hence >/.th would be 
balancing of broadcasters expensive new equipment by "°Pf ed °, ff "^2,, 0I ? y 1440 A ^E** ,he 
renewable conditional access key and advertisers credit 1920 unblanked HDTV dots and V,, times the 1440 com- 

coupons to offset consumers fee expected to be substantially P resses l ° 768 . do ^ d ' h c w I l th d ° l clock ? 

less than cable TV charges. 60 col °>, Reamer 3579545 Hz. Conversely the full NTSC 

, ,„ , „ .. . could be viewed within %ths of the HDTV width provided 

HDTV<-> rransform all 1280x720p into 640x360 of ^ m msc Uqcs m interpolated and expanded by % t0 

525TV 1080 and the 768 NTSC dots are interpolated and expaoded 

DTV Multicasting and Datacasting of ancillary info by 154. simple doubling of NTSC lines and dots is cheaper 
HDTV<->Transform all 1920xl080i into 640x360 of $5 but repeat line adds absolutely no new information. 

525 Shown separately in FIG. 2C, the full screen of HDTV is 

ATSC other 16 formats, progressive & interlace augmented by "MuIticasting"/"Datacasting" which fits into 
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the otherwise 120 blanked lines and offers distinct advan- Turning now to the WTO shown in FIG. 3, a functional 

tages over 1080i producing the same perceived resolution. detailed block diagram is shown for the set top box 10 of 

Almost 25% of the 720p transmission is specificied to FIG. 1. In this regard the TV signal may be from broadcast 

transmit ancilliary data which rate is comparable to 1.5 MHz antenna 2, from cable TV 4 or satellite which usually output 

T-l phone lines being about 25 times faster than 56 Kb/s s v ia VCR 8 to the TV set 12 a Channel 3 or 4 RF radio 

modems. As shown in FIG. 2B to squeeze the entire HDTV frequency signal (60-66 or 66-72 MHz) or video signal 

ty 9 frame into the NTSC V 3 aspect ratio then '/sth of the unlimited by the tuner. However since DTV and HDTV 

NTSC screen is typically blank on top and bottom. The 360 VCR afc not t availablc ^ yVTD video or RF output 

lines must somehow display 720p and 1080i There are feeds mtQ ^ VCR to be to ^ Qr a^yed live on me jy 

many 2x2 and 3x3 spatial filters well known in the art to do JQ scrccn Notc ^ Qf ^ w bQX 6 ^ Q ^ [t& QWn 

A 3x3 median filter compresses 1080^3 to 360 and * F tune / ( f U » ™ adv f Mt J« c ? f "»t-top Wdcvice 10 
1920/3 to 640. The median value of 3x3 arrays of dots 10 f ° ld ^ e ^ CIlse of a or fourth m thc 
typically replaces the center pixel and does not blur edges as an l lne ^T y be , ] ° w co f t „ „ . , . A . 
a running average smoothing filter does. In this case how- j ™ e C^nncl 3 or 4 RF receiver amplifier signal 20 is 
ever the median filter replaces all 9 pixels as seen in the 15 detected by 24 and tuned to a selected channel frequency 22 
readout raster as one pixel for one field. In the next field the from whicb me vj deo fe demodulated in 25 and 27 and 
3x3 window center shifts as much as the origin offset so a sync is decoded in 26. The synchronization data signals for 
new median value replaces the newer set of 9 pixels as one the GEN LOCK circuit 28 which is phase locked to the 
readout and so on. The ordered dither thereby lets the crystal controlled CLOCK GENERATOR 30 for both writ- 
eye-brain do the "error diffusion" keeping the chrominance 20 ing and reading the RAM video content. This clocks the 
and luminance in tact, as opposed to averaging and inter- Write and Read H-Count 32 which are preset by respective 
polation filters, H-sync and likewise the V-Count 34 is preset by the V-sync 

Alternatively for some images, to smooth contouring or as determined by the CPU/RAM Interface 33 and Versatile 

perception of quantizing noise the center of the 3x3 can be Video Reformat ter 35. The Origin Offset 40 to dither the 

replaced advantageously by the average of the 8 nearest 25 raster will become clear later on. Basically the RAM rows 

neighbors and be represented as one readout pixel. The and columns are addressed by counting down from preset 

subtraction of such a low pass averaging filter produces a so values until zero reloads the appropriate preset. To save 

called "Laplacian filter" that enhances edges upto a point of RAM, the HDTV can be converted going into it, so only 

objectionable noise. Because that is in morphic resonance " 640x480 pixels need to be stored. However the full HDTV 

with a similar Laplacian filter deduced to be in the visual 30 picture can be inexpensively captured in RAM so that it can 

cortex, (which overshoots perfect steps of gray) the analo- be selectively readout through the adaptive spatial filtering 

gous effect of high pass edges and/or low pass averaging can to preserve as much fidelty as possible, 
be obtained in the eye-brain itself by selectively representing The digitized Luminance (Y) and color difference signals 

one and only one of 9 pixels in each 3x3 array in each field. (Cb, Or) are latched in register 39. They are processed with 

Considering all sequences of 9 pixels, FIG. 4 zig-zag 35 the RAM Read-Mod-Write signal thru the ALU 38 arith- 

pattems show how jumping maximum distances accomplish me tic logic unit as a function of modes from CPU 31 and the 

error diffusion, whether a median, or mean or single pixel is prior (A) and present (B) values. The RAM 43 is sequenced 

representative of the 3x3 depending on the picture content thru the addresses of the pixel (H) 32 and line (V) counters 

and subjective quality. Hence the versatile video is also 34 offset each field +/-1 or 0 by 40, RAM output also feeds 

made adaptive to the distribution of spatial frequencies 40 the Sparkle/DSP filters 41 which consists of pixel and line 

contained in the MPEG coded data by the Filter Select signal delays to provide at once all video values of each instant 

from the CPU 31. pixel and its neighbors. Lookup table (LUT) 50, preloaded 

With NTSC full standard 4,2 MHz video the resolution is through Multiplexor 51 from CPU 31, converts the new 

only 448x336 even though about 480 scan lines are DTV luminance and chrominance to standard TV values to 

unblanked. The present inventions serves to overcome such 45 be generated for the purpose of optimizing the adjustment of 

limitations as will be discussed more fully. To further the TV brightness, contrast, color sharpness and tint. This 

enhance perceived resolution, a radical-2 or 41.4% boost by ALU operation can be performed for example by Texas 

2D dot interlace is possible by offset sampling by 12 MHz Instruments 74181 or else the same LUT 50 could also 

clock phase alternate lines (PAL) as shown in the lower incorporate all the ALU functions, including for example 

center of FIG. 4. As Dean Wendland of University of so transformation to RGB, Y, I, Q, HSI Hue, Saturation, 

Dortmund demonstrated, the square Fourier (frequency) Intensity for displays other than the TV set. The RAM output 

spectra is effectively rotated to a diamond so resolution on is filtered by 41 whose functional mechnanization is shown 

axis is improved by radical-2. in FIG. 4. 

Notwithstanding the foregoing TV set limitations, the The digital video output from the RAM 43 is refreshed at 

DTV transmission and present invention scan converts simi- 55 60 Hz field rate and the digital to analog converter 44 feeds 

lar 1280x720p and 1920xl080i by exploiting the space-time it to the RF modulator 46 which mixes the chroma burst and 

bandwidth in the visual perception of the viewer. sync signal before going out to the TV set thru the Input 

Furthermore, because of the video compression of DTV Output I/O RF switch 48. Note in the case of an analog TV 

transmissions several programs can be compressed with the signal coming in, the I/O switch allows it to be fed directly 

allocated channel in addition to or instead of the HDTV 60 into the TV set. This may be done automatically or by way 

format. The advantage provided by the present invention is of a manual switching device if the user knows which signal 

the feature of allowing the HDTV signal to be compressed is being transmitted. The switch may be an A/B switch in 

while maintaining the free broadcast quality of the picture which the entire set-top -box electronics integrated into one 

and also allowing renewable conditional access to the mul- Versatile Video Transformation Chip (VTQ might ulti- 

tiple broadcasts compressed into each channel. A nominal 65 mately be housed. 

user fee would offset broadcast expenses and be compen- In addition the DTV data/Not-video, from the Demod/ 

sated by TV advertisers coupons or credit 29. Decoder 27, which enables renewable conditional access 
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key, is fed to the CPU 31. This ancilliary data such as defined 
in ANSI/SMPTE 296M-1997 for 1280x720p offers almost 
the same channel capacity as dedicated 1.5 MHz phonelines 
because 25% of the 6 MHz DTV spectrum is not picture 
content. The datacasting affords high speed, low cost down- 5 
loading of information to the CPU and displays or user 
output devices besides the printer and modem shown con- 
nected to CPU 31. In addition, with the proper key decoded, 
unscrambled multicasting of low cost TV programs offers 
competitive advantages between free TV and rising cable 10 
prices. It is estimated that economies of scale to convert 
DTV for a massive market of 1.15 billion TV sets, even 
remaining about the homes with new HDTV sets, should 
enable the WTD to be so affordable initially that it will 
encourage the buyers willingness to pay for the WTD. In 15 
addition to the free portion of the allocated channel, the 
remaining portion which will be expanded due to the present 
invention allowing the broadcaster to use a format having a 
reduced bandwidth whilst maintaining an effect of HDTV 
quality, may be dedicated to subscription or pay program- 20 
ming. In such a case, the decoder can discriminate (see i.e. 
decision boxes 56) between such programming and whether 
a user is a subscriber, much the same as cable or satellite user 
systems work. 

In addition, production of standard sets will not stop 25 
immediately so they could incorportate the VVTD inside 
instead of duplicating much of the set-top-box electronics. 

Returning now to the methodology involved, conversion 
of 1920xl080i HDTV to fit properly within 360 lines of 
VGA 640x480 may be better understood by considering the 30 
WTD picture "fabric" and 3x3 median or mean (MOM) 
within Sparkle filter 41. Narenda in IEEE article had shown 
advantages of speed and circuit simplicity in finding the 
median of the median and he also pointed out how it differs 
from the true median and mean filters. The video mean, 35 
median, mode and peak-to-peak average are specifically 
different electronic video parameters but somewhat the same 
in unskewed statistics. 

For example the mean video value replacing the center of 
a 3x3 # array is called the Laplacian operator that is 40 
employed in electronic imaging for edge enhancement as 
reportedly in the visual cortex. The mean yields the least 
"mean square error" that is generally preferred because it is 
"tractable" i.e. easy to manipulate but the human eye does 
not compute mean square error as such. The 3x3 MOM 45 
exploits rationale for 1920xl080i because of simplicity of 
division by 3 and it keeps the integrity of chromaticity with 
respective luminance. 

. Because the perceived brightness is a linear response to 
density (»-Log(Relative Brightness), the average of picture 50 
elements finer than arc minute resolution defined as 20/20 
vision involves simple, scalar aritmetic. But the combination 
of HDTV picture elements chromaticity is a more complex, 
nonlinear vector process wherein mental attributes of hue 
and saturation (purity) are represented by the vector angle 55 
and magnitude respectively. Unlike averaging filters, the 
median retains proper luminance and chrominance, retains 
edge sharpness and provides noise smoothing without dis- 
tortion and without streaking due to extreme excursions of 
video amplitudes or peak-to-peak average. 60 

Median filters replace the central pixel of a NxN array 
with the median value of all values in the array. It can be 
shown that the median yields the least expected absolute 
error. Its chrominance and luminance exactly equals the 
middlemost value. More simply, the middle number of an 65 
ordered set of samples is the median. N-random samples can 
be sorted or recorded in hardware histogram as taught in 



U.S. Pat. No. 5,537,483. The extreme values do not influ- 
ence the median as they affect the mean value because the 
median value of an image histogram is the x-abscissa gray 
point at which the integral of probability density function of 
x, p(x) is half the total integral for all x. For each random 
x -value address the RAM increments by one its contents that 
is cleared after the calculation. For each x value y is the 
number of times x occurred. Compute y/N=p(x). In 3x3 y>4 
gives median x value else add up the CDF cumulative 
distribution and choose the mean or median to make the 
picture smoother or sharper. Begin again at x-0 and find the 
closest x point where the integral of p(x) upto x is the 
CDF(x) and total CDF/2 x abcissa value is the median. This 
takes time and power. 

Quicker median of median filter replaces the central pixel 
of a line of pixels with the median value and then computes 
the median of all medians which replaces the center pixel. 
The MOM however maintains HD full splendor of 1920x 
1080 by replacing every 9 pixels in 3x3 array with the 
median or mean. 

The fabric of new TV frames created by the present 
invention must be adaptive to 18 flexible -format raster 
Standards. The picture fabric is optimized by the WTD so 
that the high definition pictures SPARKLE on the old TV 
sets in the full splendor of all 1920x1080 HDTV picture 
elements (pixels) woven together to unite the motionless 
interlaced (i) fields or to alternate the raster lines of the 
"progressive" (p) noninterlaced inputs. 

The present invention enhances perceived resolution up to 
4-9 times the VGA interlaced raster output that is necessar- 
ily shortened to 640x360 because the new HDTV aspect 
ratio reduces the 480 line height 25% from 3 /4ths of the width 
to 9 /i6ths. In simplest terms, a unique 2x2 or 3x3 array 
sequence of maximum displacement vectors therein dithers 
the raster origin offset in saccade-like jerks among the 4-9 
pixels luminance (Y) and chrominance (Cb,Cr) represented 
by one readout pixel. 

After MPEG decompression, WTD DSP filters perform 
unique 4:1 or 9:1 video compression by scan converting 
down to 640x360. Another filter replaces the center HDTV 
pixel of each 3x3 array with the median value instead of 
averaging or decimating by division by 3 in each axis. In 
conjunction with the integration in space and time, the 3x3 
median of median image filter effectively smooths the higher 
definition TV without blurring the edges as nearest-neighbor 
averaging does and without jagged edges from decimated 
drop-outs and aliasing due to subsampling. Similarly the 2x2 
filters convert the 1280x720i. 

Since DTV jitter spec is less than a nanosecond, interline 
flicker from imperfect interlace is imperceptible even upon 
close viewing such that the degree of SPARKLE selected 
depends on viewing distance and nonlinear tolerance of 
complex flicker. Accordingly the CPU 31 sends Filter Select 
to 41. 

The saccade-like zigzag origin offset also provides super- 
sampling of full resolution HDTV picture to be displayed in 
time and integrated in the eye -brain. The pseudorandom 
dither sequence is ordered to minimize artifacts i.e. error 
diffusion and maximize perceived resolution. Impairments 
due to cross coupling of chrominance and luminance by 
interpolation and averaging are avoided by the WTD 
maintaining the proper chrominance with the median's true 
luminance value or that of the 4-9 supersamples. 

Where TV or VCR does not have direct video input and 
the RF tuner bandwidth is only about half the 6 MHz full 
channel bandwidth required for VGA, offset samples inter- 
lace fields of 320x180 in 2D into 640x360 of 525 frame and 
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provide a 41% enhancement by effectively rotating the 2D 
Fourier frequency response square to diamond as shown by 
Dean Wendland of University of Dortmund. Of course 
checkerboard pixel pattern becomes gray but that is very 
rare in real life. Minor flicker results. 5 

The HDTV Raster 1920xl080i interlaced fields is illus- 
trated separately and combined as the progressive 1280x 
720p frame in FIG. 4 wherein the row, column matrix 
address (r,c) in RAM corresponds to TV lines and pixel 
counters. To display HD 4-9 times more pixels on old sets 1Q 
yet minimize visible artifact, the ordered sequences of origin 
offsets are shown in FIG. 4. Each 2x2 or 3x3 block is thus 
vibrated or dithered and compressed into one pixel that 
SPARKLES with 4-9 sequential values or 4-9 medians for 
4-9 offsets. Strong pixel to pixel correlation minimizes the J5 
4-9 median differences because mathematically it is the best 
value that minimizes expected absolute deviations seen by 
the eyes whereas the average value minimizes mean-square- 
error which eyes don't compute. Since the median of even 
number of values is denned as the average of the two 
middlemost elements, the chrominance distortion discussed 
below can be avoided in the 2x2 spatial filter for 1280x720p 
by selecting the exact chrominance and luminance pixel 
value closest to the median or mean value or the most 
probable (mode). ^ 

The means to accomplish DSP spatial filtering is well 
known in the art of image processing. All 9 pixels in the 3x3 
array must be made available simultaneously so a 2 Kilobyte 
SRAM is a convenient TV-line delay when it is read out the 
next line before being rewritten. Two D -latches clocked as 3Q 
a serial shift register of one byte into another forms the pixel 
delays as shown in FIG. 4. Obviously the 3x3 hardware also 
provides the subset 2x2 array of pixels simultaneously. 

Of course design deficiencies and visual impairments 
within older TV sets will not be eliminated by the DTV or 35 
WTD and in fact some may become more objectionable to 
some small fraction of viewers. To make the A/B compari- 
son tests, the WTD includes a switch 48 of FIG. 3 to select 
the old or new, i.e. old NTSC or new ATSC HDTV or DTV 
as previously discussed. 40 

Support for the interpolation/decimation used in 41 of 
FIG. 3 beyond Nyquist sampling limits can be found in 
Fogel, L. J., "A Note of the Sampling Theorem," IRE Trans. 
Info. Theory Mar. 1, 1955 p47-48) and valid Taylor Series 
representation of any signal in terms of its time derivatives 45 
or Fourier Series in terms of sinewave frequency harmonics. 

With regard to FIG. 5 which is merely two simple 
examples, the following interpolation/decimation is per- 
formed. Banding is likely if every other NTSC line averages 
the 720p line above and below it. Curve fitting beyond two 5Q 
lines using as much as 9th order Polynomials have been 
reported. However, because of the disadvantages of inter- 
polation and decimation, the WTD attempts to employ the 
simplest filtering that relies on and exploits tuned sensitivi- 
ties in the visual perception. 5S 

Given f(kT)=df(t)/dt at t-kT, (k-0,1 A . . . ) the period of 
time between samples, 1s=l/Fso<l/Fc. Consider 9-*4 lines: 



VO/9 Vl/9 V2/9 V3/9 V4/9 V5/9 V6/9 V7/9 V8/9 V9/9 60 
*3/4 *l/4 "1/2 "1/2 *l/4 *3/4 



V0/4 



Vl/4 



V2/4 



V3/4 



V4/4 



Note the VO/9 and V9/9 are the same as VO/4 and V4/4. 65 
Since V2/4 is equidistant between V4/9 and V5/9 it is set 
equal to their average, whereas the Vl/4 and V3/4 are 



interpolated (GEOMTRICALLY weighted) values. Since 
V&-0.25 happens, to be Vitb up the gap of V&th-O.lll . . . 
between 0.222 ... and 0.333 ... it follows 

V(W)-%*V(W)+%'V(W). 

Likewise y«=0.75 happens to be %th up the gap between 
0.666 ... and 0.777 . . . such that 

V(W)-%*V(%)+tf*v(%). 

Because Vl/9 and V8/9 are totally ignored these two 
dropouts distort the decimated picture elements. By using 
the differences between all the HD DTV samples and 
differences between those differences, the 2nd derivatives 
f '(kT) provide a much more accurate characterization of the 
signals. 

Given f '(kT)=d f(t)/dt at t=kT, Ts<(%) (1/Fc). And given 
m (many) derivatives with m-th derivative being the highest, 
then Ts=l/Fs«<(^) (m+1) (1/Fc). 

The above analysis in the time domain is equivalent to 
Fourier analysis in the frequency domain and the real part of 
the so called Fast Fourier Transform (FFT) provides the 
MPEG coefficients of the DCT Discrete Cosine Transform 
in 53 of FIG, 3 used in DTV compression. They characterize 
and reconstruct video signals into 8x8 blocks, much of 
which may be zeroed to minimize redundancy subject to 
acceptable level of distortion. In general any series of video 
elements can be represented mathematically by a polyno- 
mial curve that can fit the video oscilloscope waveform as 
precisely or practical as desired. For example f (kT) gives 
straight line approximation, f '(kT) fits a parabolic shape, 
f ,f (kl) fits any cubic spline etc. 

To the extent adjacent pixels are said to be correlated they 
can be compressed effectively (eg. MPEG-2, CCIT-Group 4) 
by removing redundancy using adaptive "windowing" tech- 
niques. In the above example the sample V2/4 could be 
comprised of variable fractions (aO . . . a9) of all 9 HD DTV 
lines such as 



VO/9 Vl/9 V2/9 V3/9 V4/9 V5/9 V6/9 V7/9 V8/9 V9/9 
*a0 *al *a2 *a3 *a4 'a5 *a6 *a7 »a8 *a9 
+ 

-V2/4 

but in practice 3x3, 5x5 or 7x7 windows aka spatial filters 
are good enough and often are more easily implemented in 
real time as if spatial frequencies of the video image are 
separable in the horizontal and vertical direction. Note 3x3 
# nearest neighbor average is in morphic resonance with 2nd 
derivative Laplacian operator (del-squared VV) in the visual 
cortex. 

For NTSC only a common GEN LOCK SYNC CHIP runs 
at 4 times color subcarrier such that when divided down by 
455 the 2Fh is produced which produces interlace by further 
division by 525. Although the picture elements are not 
square the 768 dots per width correspond to a video band- 
width of 7.2 MHz or about twice VCR resolution. 

It is often desireable to transform RGB, Y,I,Q or Y, Cb, Cr 
values that are interdependent, to the subjective HSV Hue, 
Saturation, Value color space because the HSV attributes are 
supposedly independent. A red and green pixel should not 
produce a yellow as averaging would, except when the TV 
color triads are deliberately less than the arc minute angular 
subtense defined as 20/20 vision. 

The foregoing filtering effectiveness and operability is 
supported by the following publications: Optimum Signal 
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Processing by (Prof) Sophocles Orfanidis at Rutgers SUNJ, 
Discrete-Time Signal Processing by Oppenheim & Schafer, 
IEEE Special Issue on Digital Image Processing July 1972 
and most recently Digital Television Fundamentals by 
Robin & Poulin and standards mentioned above. 

Thus the present invention realizes the aforenoted 
objectives, advantages and features, and although the pre- 
ferred embodiments have been disclosed and described in 
detail herein, its scope should not be limited thereby but 
rather its scope should be determined by the appended 
claims. 

What is claimed is: 

1. A versatile video transformation device that allows for 
conventional television sets to utilize input signals with 
HDTV, DTV or formats with enhanced data 
communications, compression, aod improved picture quality 
perception comprising: 

means for decoding said signal and scan converting them 
into an enhanced visual image from standard TV raster 
format and which comprises; 

means for spatial and temporal filtering said input signals 
by adaptive sampling said input signals into selectable 
adjacent arrays, determining a selected value for each 
array, said selected value being selected from one of the 
median, mean (average) of the input signals and a 
sample value of each array about a center pixel defined 
with respect to the raster origin which is offset from a 
previous raster origin in the respective arrays, and 
displaying at least one single pixel in the array depen- 
dent on said selected value; and 

wherein field sequential arrays are filtered and said 
selected value for each array that is used to readout a 
video value in each array is determined based upon the 
decoded picture content, image motion, and respective 
origins that are offset with respect to each other in an 
ordered dither fashion for error diffusion. 

2. The invention in accordance with claim 1 wherein the 
digital signal is in one of a 1920xl080i, 1280x720p HDTV 
format. 

3. The invention in accordance with claim 1 wherein the 
scan conversion temporal and spatial filtering is without 
color distorted crosscoupling of chrominance and luminance 
so as to sparkle on lower resolution TV sets while substan- 
tially maintaining the input detail subject to viewers com- 
plex spatial/temporal flicker tolerance. 

4. The invention in accordance with claim 1 wherein the 
ordered dithering of raster origin offsets is provided in a 
zigzag fashion within a 2x2 array of filtered pixels for scan 
conversion into one pixel readout so as to sparkle on lower 
resolution TV sets while substantially maintaining the input 
detail subject to viewers complex spatial/temporal flicker 
tolerance. 

5. The invention in accordance with claim 1 wherein the 
ordered dithering of raster origin offsets is provided in a 
zigzag fashion with 3x3 array of 9 pixels displayed at one 
pixel so as to sparkle on lower resolution TV sets while 
substantially maintaining the input detail subject to viewers 
complex spatial/temporal flicker tolerance. 

6. The invention in accordance with claim 1 which further 
comprises renewable conditional access to demultiplex mul- 
ticast programs and ancillary data compressed in packets 
within one DTV channel allocation. 

7. The invention in accordance with claim 6 which 
includes means to indicate consumer reimbursement of the 
use fee of conditional access channels by one of advertisers* 
coupons and credit. 

8. A method of providing digital scan conversion for 
image signals having digital signal formats comprising the 
steps of: 
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providing adaptive image acquisition means to minimize 
grainy appearance of electronic noise frozen with the 
signal; 

providing adaptive thresholding of image subtraction 
s means to detect image motion and thereby limit smear- 
ing; and 

providing digital versatile video spatial/temporal filtering 
means responsive to decoded picture parameter signals 
and automatically synchronized to formats of vertical 
10 and horizontal sync signals associated with the input 
video and phase locked to picture element clocks from 
a gen-lock circuit which also serve to clock counters 
addressing RAM random access memory input and 
read out, 

15 9. The method in accordance with claim 8 which includes 
providing PC-TV-DSP convergence means which enables 
standard TV sets to become compatible with DTV, HDTV 
multicasting, datacasting and formats of PC monitors and 
hard copy printers. 

20 10. The method in accordance with claim 8 which 
includes the step of incorporating electronics to perform the 
method within TV sets to minimize redundancy of hardware 
and exploit frame memory required therein for image redun- 
dancy removal by compression. 

25 11. The method in accordance with claim 8 which 
includes providing an integrated circuit chip which performs 
the steps set forth therein. 

12. The invention in accordance with claim 1 wherein said 
video transformation device is incorporated in an integrated 

30 circuit chip. 

13. A method of transforming a digitally formatted tele- 
vision signal into a form utilizable by a conventional tele- 
vision comprising the steps of: 

spatial and temporal filtering said digital TV signals by 
35 adaptive sampling said DTV into selectable adjacent 
arrays, determining a selected value for each array 
based on a sample value of the respective arrays, the 
median, or mean (average) of the signals for each array 
about a center pixel defined with respect to the raster 
40 origin which is offset from a previous raster origin for 
displaying at least one single pixel in the array based on 
said selected value; and 
providing field sequential arrays which are filtered and 
said selected value for each array that is used to provide 
45 a readout video value in each array which is determined 
based upon the picture content, image motion, and 
respective origins that are offset with respect to each 
other in an ordered dither fashion for error diffusion. 

14. The method in accordance with claim 13 wherein the 
50 digital signal is selected from signals having one of a 

1920xl080i and 1280x720p format. 

15. The method in accordance with claim 13 wherein scan 
conversion temporal and spatial filtering is provided without 
color distorted crosscoupling of chrominance and luminance 

55 so as to provide sparkle on lower resolution TV sets while 
substantially maintaining the input detail subject to viewers 
complex spatial/temporal flicker tolerance. 

16. The method in accordance with claim 13 wherein the 
ordered dithering of raster origin offsets is provided in a 

60 zigzag fashion within a 2x2 array of filtered pixels for scan 
conversion into one pixel readout so as to provide sparkle on 
lower resolution TV sets while substantially maintaining one 
of a 1280x720p and an input detail subject to viewers 
complex spatial/temporal flicker tolerance. 

65 17. The method in accordance with claim 13 wherein the 
ordered dithering of raster orjgin offsets is provided in a 
zigzag fashion with 3x3 array for median or mean filtering 
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of 9 pixels into 9 medians displayed at one pixel so as to 
provide sparkle on lower resolution TV sets while substan- 
tially maintaining one of a 1920xl080i and an input detail 
subject to viewers complex spatia/temporal flicker tolerance. 

18. The method in accordance with claim 13 which 
further comprises the step of providing renewable condi- 
tional access to demultiplex multicast programs and 
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ancillary data compressed in packets within one DTV chan- 
nel allocation. 

19. Hie method in accordance with claim 13 which 
includes means to indicate consumer reimbursement of the 
use fee of conditional access channels by one of advertisers' 
coupons or and credit 
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